Selecting, smoothing, and deriving measures from motion tracking, and merging with acoustics and annotations in Python

Wim Pouw (wim.pouw@donders.ru.nl)

Info documents

• This python coding module shows some basic data wrangling procedures that are often required when analyzing motion, sound, for events bounded by annotations. These procedures include merging data streams, deriving 1-dimensional measures from high dimensional motion tracking data, accounthing for different sampling rates, and smoothing the data. We also show some simple applications once we have a multimodal dataset prepared, in the form of a simple peak analysis.

• location Repository: https://github.com/WimPouw/envisionBOX_modulesWP/tree/main/MultimodalMerging

• location Jupyter notebook: https://github.com/WimPouw/envisionBOX_modulesWP/blob/main/MultimodalMerging/MergingMultimodal_inPython.ipynb

• citation: Pouw, W. (2023). Selecting, smoothing, and deriving measures from motion tracking, and merging with acoustics and annotations. [the day you viewed the site]. Retrieved from: https://envisionbox.org/embedded_MergingMultimodal_inPython.html

Background

In multimodal analysis you will often encounter the situation that you have signals that are continuous but sampling at different rates, and such signals then also need to be related to qualitative codings such as ELAN gesture annotations, or trial information of your experiment. It is sometimes convenient to have everything you need in one time series file, so you can apply your multimodal analysis easily. A lot of the initial steps of even beginning to do some quantitative multimodal recording and analysis are covered in Pouw, Trujillo, Dixon (2020); here we provide some basic data wrangling steps that are often required to set up your dataset (e.g., merging data, smoothing data, extracting kinematic variables).

## Video Tutorial (from envision bootcamp 2021)
#from IPython.display import YouTubeVideo
#YouTubeVideo('https://youtu.be/0uAa10VbVes?list=PL8vb5RWWmZQCWjjnP2HQT4kPI2Q5XKB0D', width=400, height=300)
# We will update this after the summer school

Set up folders and check data formats

For this module we will only demonstrate the steps for the cartoon retelling example that we have in our multimedia samples. For this sample we have already generated

• A) a motion tracking time series for a bunch of body keypoints sampling at 30Hz
• B) an amplitude envelope time series of speech which was sampled at 100Hz
• C) Then we also have annotations of the dominant hand (right handed gestures) that this person produced.

So here we show a way to merge A, B, and C, in a way that is convenient for further analysis. Lets first identify the relevant files and set the relevant folders.

import os            #folder designating
import pandas as pd  #data wrangling and data framing

curfolder = os.getcwd() #get current working folder
#Load in the motion tracking data
MT = pd.read_csv(curfolder + "/MotionTracking/video_cartoon.csv")
#Load in the amplitude envelope
ENV = pd.read_csv(curfolder + "/AmplitudeEnvelope/audio_cartoon_ENV.csv")
#load in the relevant annotations                                            
ANNO = pd.read_csv(curfolder + "/MultimediaAnnotations/annotations_cartoon.csv")
#This is the folder where your merged output is saved                                                 
outputfolder = curfolder + "/output/"

print("The data we have are the motion tracking data with " + str(MT.shape[1]) + 
      " columns and " +  str(MT.shape[1]-1) + "body keypoints, sampling at" + 
      str(round(MT['time'].diff().median())) + " ms intervals")
MT.head()

The data we have are the motion tracking data with 133 columns and 132body keypoints, sampling at33 ms intervals

	time	X_NOSE	Y_NOSE	Z_NOSE	visibility_NOSE	X_LEFT_EYE_INNER	Y_LEFT_EYE_INNER	Z_LEFT_EYE_INNER	visibility_LEFT_EYE_INNER	X_LEFT_EYE	...	Z_RIGHT_HEEL	visibility_RIGHT_HEEL	X_LEFT_FOOT_INDEX	Y_LEFT_FOOT_INDEX	Z_LEFT_FOOT_INDEX	visibility_LEFT_FOOT_INDEX	X_RIGHT_FOOT_INDEX	Y_RIGHT_FOOT_INDEX	Z_RIGHT_FOOT_INDEX	visibility_RIGHT_FOOT_INDEX
0	0.000000	-0.063733	0.502805	-0.395698	0.999921	-0.083962	0.538291	-0.383986	0.999944	-0.083089	...	0.130854	0.192507	0.018961	-0.489134	-0.100693	0.418703	-0.002238	-0.516333	0.090475	0.193644
1	33.333333	-0.043211	0.502240	-0.394696	0.999912	-0.071669	0.535697	-0.384011	0.999936	-0.070713	...	0.119296	0.194849	0.034104	-0.402429	0.040178	0.397703	-0.005346	-0.358187	0.092969	0.195986
2	66.666667	-0.020163	0.503504	-0.386350	0.999909	-0.051884	0.535895	-0.380524	0.999936	-0.050845	...	0.115370	0.193730	0.053036	-0.289042	0.061571	0.381764	-0.005315	-0.230478	0.111993	0.196811
3	100.000000	-0.008440	0.533132	-0.351903	0.999891	-0.043266	0.560953	-0.352411	0.999932	-0.042234	...	0.198556	0.207998	0.045575	-0.337201	0.151154	0.368794	0.006082	-0.334947	0.215761	0.206893
4	133.333333	0.002767	0.559002	-0.262275	0.999872	-0.036010	0.584854	-0.269395	0.999932	-0.035020	...	0.222347	0.214859	0.037744	-0.280906	0.162776	0.364183	0.015523	-0.267834	0.253445	0.212417

5 rows × 133 columns

print("The data for the amplitude envelope have " + str(ENV.shape[1]) + 
      " columns, sampling at " + 
      str(round(ENV['time_ms'].diff().median())) + " ms intervals")
ENV.head()

The data for the amplitude envelope have 2 columns, sampling at 10 ms intervals

	time_ms	env
0	10.0	0.0
1	20.0	0.0
2	30.0	0.0
3	40.0	0.0
4	50.0	0.0

print("And the annotations of gestures of the right hand, with begintime, endtime, and annotation information. In total we have " + str(ANNO.shape[0]) + " annotations, and three columns.")
ANNO.head()

And the annotations of gestures of the right hand, with begintime, endtime, and annotation information. In total we have 36 annotations, and three columns.

	Begin Time - msec	End Time - msec	annot_gesture_right
0	7775	8255	BEAT
1	12885	13705	REP
2	14265	14835	BEAT
3	14975	15655	REP
4	16645	17355	REP

Select MT and merge with acoustic envelope data

Lets not merge all data. For the motion tracking output we generated for the cartoon video we are now only interested in some specific body part; say we are interested in the right hand index finger traces only. Lets select them first.

selection = ["time", "X_RIGHT_INDEX", "Y_RIGHT_INDEX" ,"Z_RIGHT_INDEX"] #concatenate some variable names in a list called "selection"
MTs = MT[selection]

This selection of the motion tracking data we want to then align with the acoustic data. We use pandas merge function for this, and we align the acoustic and motion tracking data based on their common information (namely time in milliseconds). We do we want to make sure that we keep information from both objects, instead of only aligning when one and the other has a value (we therefore set how = "outer" as this is callled an outer join where all rows of both dataframes will be merged with ).

# before merging lets rename the column time_ms in env to time so that they get properly merged into one column
ENV = ENV.rename(columns={'time_ms': 'time'})

# Merge the data frames using an outer join
merged = pd.merge(MTs, ENV, on ='time', how='outer')
# Sort the dataframe by Time1 and Time2
merged = merged.sort_values(by=['time']) #lets order the dataframe along the time_ms
# Display the head of the merged data frame
print(merged.head())

           time  X_RIGHT_INDEX  Y_RIGHT_INDEX  Z_RIGHT_INDEX  env
0      0.000000       0.230409      -0.011444      -0.358735  NaN
4641  10.000000            NaN            NaN            NaN  0.0
4642  20.000000            NaN            NaN            NaN  0.0
4643  30.000000            NaN            NaN            NaN  0.0
1     33.333333       0.227738      -0.031909      -0.332289  NaN

We can see that while we have ordered and aligned the two objects in a single merge object, we have a lot of empty Non-Applicable (NaN) rows. This is because at the exact times the sample is taken for the amplitude envelope there is not a sample for motion tracking. The solution is to linearly interpolate and upsample your data. We will do this by approximating for each NaN for motion tracking sample what its value would be given that it is at time x and we know the values at a particular time before and after. Pandas has a native function for this. By stating which series you want to interpolate NaN's for (e.g., X_RIGHT_INDEX), given some information about the time (x = time). We can leave NaN's that we aren't able to interpolate, e.g., if your merged time series ends with NaN's we cant interpolate because we dont have values we can use for interpolation.

# Perform interpolation for specified columns
merged['X_RIGHT_INDEX'] = merged['X_RIGHT_INDEX'].interpolate(method='linear', x=merged['time'])
merged['Y_RIGHT_INDEX'] = merged['Y_RIGHT_INDEX'].interpolate(method='linear', x=merged['time'])
merged['Z_RIGHT_INDEX'] = merged['Z_RIGHT_INDEX'].interpolate(method='linear', x=merged['time'])
# Alternatively, perform interpolation for all columns
# merged.iloc[:, 2:5] = merged.iloc[:, 2:5].interpolate(method='linear', x = merged['time_ms'])

We are now almost done with the merging of acoustics and motion tracking. First, we should note, that there is an important reason why we choose to upsample the motion tracking data from 30Hz to 100Hz, and this is because we would be loosing information if we would downsample the amplitude envelope from 100Hz to 30Hz.

Now that we have upsampled the motion tracking data, we can just go ahead and only keep information where we both have info from the amplitude envelope and info from motion tracking; this will yield a time series object with steadily samples at 100Hz with original data points for the amplitude envelope, and interpolated and upsampled values for the motion tracking (we discard the original samples from the motion tracking).

# Remove rows with NA values in the 'env' column
merged = merged.dropna(subset=['env'])

# Remove trailing NA values
merged = merged.dropna(subset=['X_RIGHT_INDEX'])

# make sure that the indices are set from 0 to n
merged = merged.reset_index()

# only keep the following columns
tokeep = ['time', 'X_RIGHT_INDEX', 'Y_RIGHT_INDEX', 'Z_RIGHT_INDEX', 'env']
# Display the updated DataFrame
merged = merged[tokeep]

merged.head() #lets look at what we created so far

	time	X_RIGHT_INDEX	Y_RIGHT_INDEX	Z_RIGHT_INDEX	env
0	10.0	0.229741	-0.016560	-0.352124	0.0
1	20.0	0.229074	-0.021677	-0.345512	0.0
2	30.0	0.228406	-0.026793	-0.338901	0.0
3	40.0	0.219933	-0.022496	-0.323767	0.0
4	50.0	0.212127	-0.013083	-0.315245	0.0

Inspecting data, deriving some motion tracking measures, and applying smoothing

So we now have a 'merged' data file that contains fully time aligned data about movement and acoustics. Our first multimodal time series object! Lets do some plotting of the amplitude envelope against the position traced we have of the index finger for an arbitrary 5 second sample.

import plotly.graph_objects as go
from plotly.subplots import make_subplots

# Assuming you have a DataFrame named 'merged', were taking a subsample from it
sample = merged[merged['time'].between(13000, 16000)]

# Create the first plot
fig1 = go.Figure()
fig1.add_trace(go.Scatter(x=sample['time'], y=sample['env'], mode='lines'))

# Create the second plot
fig2 = go.Figure()
fig2.add_trace(go.Scatter(x=sample['time'], y=sample['Z_RIGHT_INDEX'], name='z', mode='lines', line=dict(color='red')))
fig2.add_trace(go.Scatter(x=sample['time'], y=sample['Y_RIGHT_INDEX'], name='y', mode='lines', line=dict(color='gold')))
fig2.add_trace(go.Scatter(x=sample['time'], y=sample['X_RIGHT_INDEX'], name='x', mode='lines', line=dict(color='blue')))

# Create subplots
fig = make_subplots(rows=2, cols=1, subplot_titles=('Envelope', 'x, y, z displacement'))

# Add the plots to the subplots
fig.add_trace(fig1.data[0], row=1, col=1)
fig.add_trace(fig2.data[0], row=2, col=1)
fig.add_trace(fig2.data[1], row=2, col=1)
fig.add_trace(fig2.data[2], row=2, col=1)

# Update the subplots layout
fig.update_layout(showlegend=False)

# Display the subplots
fig.show()

Smoothing

One thing that you will run into when using motion tracking data, especially when using video based motion tracking data, is that you will have noise-related jitter in your time series. At some times such noise maybe minimal, e.g., when using very accurate device-based motion tracking devices. But in other cases, you will see that there are sudden jumps or kinks from time point to time point due to tracking inaccuracies (that can be caused by occlusions, or not ideal lighting, camara position changes, etc.).

It is good therefore to apply some smoothing to the position traces of your motion tracking data, as well as any derivatives that are approximated afterwards (e.g., 3D speed, vertical velocity). You can for example apply a low-pass filter, whereby you try to only allow fluctuations that have a slow frequency change (gradual changes from point to point) so as to filter out (i.e., reduce the amplitude of) the jitter that occurs at very high frequencies (because they result in sudden changes from point to point). Note that when using low-pass filters there can be some time shift, so in that case it is good to undo that shift by running the smoothing forwards and backwards (we do this by using filtfilt); this undoing of distortions in time is called a “zero-phase” low-pass filter. Applying zero-phase low-pass filters is important if you care about precise temporal precision relative to some other timeseries for example (e.g., acoustics).

We can also use a different kind of smoothing filter where the cut-off frequency is not strictly defined such as a running weighted average or Gaussian filter, such that sudden changes in the time series are smoothed out by relating them in some weighted way to the neighboring data points. Below we use such a neighbor averaging filter called a Gaussian filter.

So we provide two filters, and we also show some differences in settings. The amount of possible filters is immense. To pick one filter without becoming an expert on filters (but see [2] chapter 4 for a really nice resource to read up on this), you can try a few and then assess it with your own data and see how a filter is capturing the variability you are interested in (for example by checking against the video of the movements). If you care about small amplitude fluctuations in your time series that might have to do with jerky movement, a heavy filter can potentially destroy this variability. A too weak of a filter might leave the signal riddled with noise, which may look acceptable, but can cascade into dramatic noisy estimates when taking for example derivatives as they amplify power at faster frequencies (e.g., speed -> acceleration -> jerk) (see e.g., [3] chapter 3 for a nice discussion on this). There are also ways to empirically assess what high frequency noise is in the data, given that noise generally has a particular random structure (see [2]).

As an excercise, try to play with the parameters of the smoothing functions a little, and zoom in on the graph, to see what effects it has.

import numpy as np
from scipy.signal import butter, filtfilt
from scipy.ndimage import gaussian_filter1d

# Butterworth filter function
    #take some time series, apply the filter, then return it 
def butter_it(x, sampling_rate, order, lowpass_cutoff):
    nyquist = sampling_rate / 2
    cutoff = lowpass_cutoff / nyquist  # Normalized frequency
    b, a = butter(order, cutoff, btype='low')
    filtered_x = filtfilt(b, a, x)
    return np.asarray(filtered_x, dtype=np.float64)

# set Gaussian filter 
guassian_sigma = 2.0  # Standard deviation

# make a new throwaway dataframe
tempmerge = merged

# we can see more dramatic effects of the smoothing when we add some noise to the time series
    # Add Gaussian noise
mean = 0  # Mean of the Gaussian distribution
std_dev = 0.001  # Standard deviation of the Gaussian distribution
noise = np.random.normal(mean, std_dev, size=len(tempmerge['Z_RIGHT_INDEX']))
tempmerge['Z_RIGHT_INDEX']  = tempmerge['Z_RIGHT_INDEX'] + noise

# Add a heavy filter with lowpass cutoff 10
tempmerge['Z_RIGHT_INDEXlowpass10'] = butter_it(tempmerge['Z_RIGHT_INDEX'], sampling_rate=100, order=1, lowpass_cutoff=10)

# Add a milder filter with lowpass cutoff 30
tempmerge['Z_RIGHT_INDEXlowpass30'] = butter_it(tempmerge['Z_RIGHT_INDEX'], sampling_rate=100, order=1, lowpass_cutoff=30)

# Also add a guassian filter (sigma is the standard deviation of the guassian)
tempmerge['Z_RIGHT_INDEXguassianfilt'] = gaussian_filter1d(tempmerge['Z_RIGHT_INDEX'], sigma = 2)

# Create the plot using Plotly Express (px)
# Create the second plot 
fig3 = go.Figure()
fig3.add_trace(go.Scatter(x=tempmerge['time'], y=tempmerge['Z_RIGHT_INDEX'], name='z raw', mode='lines', line=dict(color='black')))
fig3.add_trace(go.Scatter(x=tempmerge['time'], y=tempmerge['Z_RIGHT_INDEXlowpass10'], name='z smooth 10 Hz', mode='lines', line=dict(color='red')))
fig3.add_trace(go.Scatter(x=tempmerge['time'], y=tempmerge['Z_RIGHT_INDEXlowpass30'], name='z smooth 30 Hz', mode='lines', line=dict(color='gold')))
fig3.add_trace(go.Scatter(x=tempmerge['time'], y=tempmerge['Z_RIGHT_INDEXguassianfilt'], name='z smooth Guassian', mode='lines', line=dict(color='green')))

# Create subplots
fig3.show()

So in the above figures you can zoom in see the differences between different kind of filter settings. Our Mediapipe tracking sample is already quite smooth, but in our experience video based motion tracking can be quite jittery and smoothing is crucial then. For now we will apply for all our motion tracking data the 2nd order zero-phase butterworth filter at a frequency cutoff of 20Hz.

#apply a butterworth filter to the following position traces
cs = ['X_RIGHT_INDEX', 'Y_RIGHT_INDEX','Z_RIGHT_INDEX']
merged[cs] = merged[cs].apply(lambda x: butter_it(x=x,sampling_rate=100, order=2, lowpass_cutoff=20))

Computing speed and acceleration (and smoothing again)

We now already have only for one body joint three variables to describe its position. Sometimes we are only interested in a 1-dimensional signal, such as rate of position change in a particular dimension (e.g., vertical velocity), or the rate of change of position in any direction (3D speed), or the change of 3D speed (acceleration). Below some simple examples of how to compute this from your initial position data using forward differientiation.

# function that differientates and butterworth filters the speed vector
def derive_it(x):
    x = np.concatenate(([0], np.diff(x)))
    x= butter_it(x, sampling_rate=100, order=2, lowpass_cutoff=20)
    return x

# function to calculate the speed vector
def get_speed_vector(x, y, z, time_millisecond):
    # calculate the Euclidean distance from time point x to time point x+1, for 3 dimensions
    speed = np.concatenate(([0], np.sqrt(np.diff(x) ** 2 + np.diff(y) ** 2 + np.diff(z) ** 2)))
    speed = butter_it(speed, sampling_rate=100, order=2, lowpass_cutoff=20)

    # scale the speed vector so that we express it units change per second change
    time_diff = np.mean(np.diff(time_millisecond)) / 1000
    speed = speed / time_diff
    return speed

# function to scale the time series
def sc_it(x):
    return (x - np.mean(x)) / np.std(x, ddof=0)

#make a new variable in a pandas dataframe
merged['speed'] = get_speed_vector(merged['X_RIGHT_INDEX'], merged['Y_RIGHT_INDEX'], merged['Z_RIGHT_INDEX'], merged['time'])
merged['vertical_velocity'] = derive_it(merged['Y_RIGHT_INDEX'])/np.mean(np.diff(merged['time']))
merged['acceleration'] = derive_it(merged['speed'])
merged['jerk'] = derive_it(merged['acceleration'])

cs = ['speed', 'vertical_velocity', 'acceleration', 'jerk']
merged[cs] = merged[cs].apply(lambda x: sc_it(x))


# Create the plot using Plotly Express (px)
# Create the second plot
fig4 = go.Figure()
fig4.add_trace(go.Scatter(x=merged['time'], y=merged['speed'], name='speed', mode='lines', line=dict(color='black')))
fig4.add_trace(go.Scatter(x=merged['time'], y=merged['vertical_velocity'], name='vertical velocity', mode='lines', line=dict(color='red')))
fig4.add_trace(go.Scatter(x=merged['time'], y=merged['acceleration'], name='acceleration', mode='lines', line=dict(color='gold')))
fig4.add_trace(go.Scatter(x=merged['time'], y=merged['jerk'], name='jerk', mode='lines', line=dict(color='green')))
# show only a portion of the plot for the x axis
fig4.update_xaxes(range=[1800, 2600])
fig4.show()

Note. Notice for the figure that vertical velocity can go positive (upward rate of motion) or negative (downward rate of motion); it is a vector as the values have a direction. This is similar for acceleration (positive = acceleration, negative = deceleration) and jerk (positive = increasing acceleration, negative = decreasing acceleraiton). Speed is a scalar value and does not go negative.

Adding annotations and saving data

We now have merged the acoustic data with an upsampled selection of the motion tracking data, and we have derived some kinematic variables and added them to our time series dataset. The only thing that is left, is to merge the annotations with this dataset. The following code loads in the annotations (with begintime, endtime, and annotation) in the time series data. We also add an identifier to the gesture coding, such that each gesture event gets a unique identifier.

# rename the columns 1 to 3 with
# Rename the columns 1 to 3
ANNO.columns = ['begintime', 'endtime', 'gesture_type']

#make a new gesture ID for each annotation using the original begin and end times
tokeep = ['begintime', 'endtime']

# ANNOID is a dataframe with the original begin and end times
ANNOID = ANNO[tokeep]

#make a list of gesture IDs
ids = []
for i in range(0, len(ANNO)):
    id = "GID_" + str(i)
    ids.append(id)
# add the list of gesture IDs to the dataframe
ANNOID['IDs'] = ids

# this function loads in annotations and the original time of the timeseries dataframe, and returns annotations for the time series dataframe
def load_in_event(time_original, anno):
    output = np.full(len(time_original), np.nan, dtype=object)  # Initialize output array with NaN values
    for i in range(len(anno)):
        # Assign the gesture type if the time is between the begin and end time of the annotation (third column of your annotation object)
        output[(time_original >= anno.loc[i, 'begintime']) & (time_original <= anno.loc[i, 'endtime'])] = anno.iloc[i, 2]
    return output

# apply the function to the merged dataframe
merged['gesture_type'] = load_in_event(merged['time'], ANNO)
merged['gesture_ID'] = load_in_event(merged['time'], ANNOID)

#lets save the data now we have everything merged
merged.to_csv(outputfolder+'merged_python.csv', index=False)

Some applications

So this new merged dataset is very flexible, in that we can for example, loop through relevant events and extract information from it. For example in the below lines of code, we ask for each gesture what the peak speed was, and then also the maximum in the amplitude envelope, and their timings. We then calculate the time difference between the speech and the movement peaks (synchrony). We then provide a smoothed frequency distribution to see how the timing distribution per gesture type.

peak analysis

import plotly.graph_objects as go
from plotly.subplots import make_subplots
import plotly.figure_factory as ff

# Example plot
    # pick an example gesture ID
pickedexample_indices = merged[merged['gesture_ID'] == merged['gesture_ID'].unique()[2]].index
    # pick the time series chunck for that gesture ID
sample_ts = merged.loc[pickedexample_indices]
    # pick the max speed value and time, same for the max env value and time
maxspeed_value = sample_ts['speed'].max()
maxspeed_time = sample_ts.loc[sample_ts['speed'].idxmax(), 'time']
maxenv_value = sample_ts['env'].max()
maxenv_time = sample_ts.loc[sample_ts['env'].idxmax(), 'time']
    # make the plot
fig = make_subplots(rows=1, cols=2)
    # add the traces
fig.add_trace(
    go.Scatter(x=sample_ts['time'], y=sample_ts['speed'], mode='lines', name='speed'),
    row=1, col=1
)
fig.add_trace(
    go.Scatter(x=[maxspeed_time], y=[maxspeed_value], mode='markers', marker=dict(color='red', size=8), name='max speed'),
    row=1, col=1
)
fig.add_vline(x=maxenv_time, line_color='purple', line_dash='dash', row=1, col=1)
fig.add_vline(x=maxspeed_time, line_color='red', line_dash='dash', row=1, col=1)

fig.add_trace(
    go.Scatter(x=sample_ts['time'], y=sample_ts['env'], mode='lines', name='env'),
    row=1, col=2
)
fig.add_trace(
    go.Scatter(x=[maxenv_time], y=[maxenv_value], mode='markers', marker=dict(color='purple', size=8), name='max env'),
    row=1, col=2
)
# add vertical lines that match the time of each peak
fig.add_vline(x=maxenv_time, line_color='purple', line_dash='dash', row=1, col=2)
fig.add_vline(x=maxspeed_time, line_color='red', line_dash='dash', row=1, col=2)

fig.update_xaxes(title_text='time', row=1, col=1)
fig.update_yaxes(title_text='speed', row=1, col=1)
fig.update_xaxes(title_text='time', row=1, col=2)
fig.update_yaxes(title_text='env', row=1, col=2)
fig.update_layout(title='Example Plot', showlegend=False)

# Apply routine to the whole dataset
maxs = []
maxenv = []
maxst = []
maxenvt = []
g_type = []
g_ID = []

for id in merged['gesture_ID'].unique():
    if pd.notnull(id):
        indices = merged[merged['gesture_ID'] == id].index
        slicemerged = merged.loc[indices]
        maxs.append(slicemerged['speed'].max())
        maxst.append(slicemerged.loc[slicemerged['speed'].idxmax(), 'time'])
        maxenv.append(slicemerged['env'].max())
        maxenvt.append(slicemerged.loc[slicemerged['env'].idxmax(), 'time'])
        g_type.append(slicemerged['gesture_type'].unique()[0])
        g_ID.append(slicemerged['gesture_ID'].unique()[0])

mag_D = pd.DataFrame({
    'maxs': maxs,
    'maxenv': maxenv,
    'maxst': maxst,
    'maxenvt': maxenvt,
    'g_type': g_type,
    'g_ID': g_ID
})
mag_D['synchrony'] = mag_D['maxst'] - mag_D['maxenvt']

# make a density distribution
fig_hist = ff.create_distplot([mag_D['synchrony']], ['synchrony'], show_hist=False)
fig_hist.update_layout(title='Smoothed Density Distribution', xaxis_title='synchrony between peak speed and peak envelope', yaxis_title='Density')

fig.show()
fig_hist.show()

References

1. Pouw, W., Trujillo, J. P., & Dixon, J. A. (2020). The quantification of gesture–speech synchrony: A tutorial and validation of multimodal data acquisition using device-based and video-based motion tracking. Behavior research methods, 52(2), 723-740.
2. Challis, J. H. (2020). Experimental Methods in Biomechanics. Springer Nature.
3. Winter, A. (2009). Biomechanics and Motor Control of Human Movement. Wiley & Sons.